Self-adaptation and mutual adaptation for distributed scheduling in benevolent clouds

1College of Information Science and Engineering, Hunan University, Changsha, Hunan, China 2College ofMathematics and Computer Science, HunanNormal University, Changsha, Hunan, China 3Department of Electrical and Computer Engineering, North Dakota State University, Fargo, ND, USA 4Department of Computer Science, State University of NewYork, New Paltz, NY, USA Correspondence Zheng Xiao, College of Information Science and Engineering, Hunan University, Changsha, Hunan 410082, China. Email: [email protected] Summary Joint service involving several clouds is anemerging formof cloudcomputing. Inhybrid clouds, the schedulers within 1 cloud must not only self-adapt to the job arrival processes and the workload but also mutually adapt to the scheduling polices of other schedulers. However, as a combinatorial optimization problem, scheduling is challenged by the adaptation to those dynamics and uncertain behaviors of the peers. This article studies the collaboration among benevolent clouds that are cooperative in nature and willing to accept jobs from other clouds. We take advantage of machine learning and propose a distributed scheduling mechanism to learn the knowledge of jobmodel, resource performance, and others’ policies.Without explicit modeling and prediction, machine learning guides scheduling decisions based on experiences. To examine the performance of our approach,we conducted simulation using the SP2 jobworkload log of the SanDiego Supercomputer Center under a test bed based on agent-based systems—SWARM. The results validate that our approach has much shorter mean response time than 5 typical dynamic scheduling algorithms—opportunistic load balancing, minimum execution time, minimum completion time, switching algorithm, and k-percent best. A better collaboration in hybrid cloud is achieved by full adaptation.